Pilot Willingness to Take Off Into Marginal Weather, Part II: Antecedent Overfi tting With Forward Stepwise Logistic Regression

Adverse weather is the leading cause of fatalities in general aviation (GA). In prior research, influences of ground visibility, cloud ceiling height, financial incentive, and personality were tested on 60 GA pilots' willingness to take off into simulated adverse weather. Results suggested that pilots did not see " weather " as a monolithic cognitive construct but, rather, as an interaction between its separate factors. However, methodological issues arose during the use of logistic regression in modeling the effect of 60+ candidate predictors on the outcome variable of takeoff into adverse weather. It was found quite possible to obtain false " significance " for models comprised merely of random numbers, even when the number of model predictors was limited to a conventional 1/10. Therefore, Monte Carlo simulations were used to derive unbiased estimates of model significance and R 2 values. Research in correction for this case/candidate predictor ratio effect is relatively new and noteworthy, particularly in the social sciences. It was given the name " antecedent overfitting " to contrast with the more commonly known " postcedent " type, which is based on a small case/model predictor ratio. of Pennsylvania, who reviewed this manuscript. we can actually overfi t the model by juggling the β (beta) coeffi cients in the exponent, β 0 + β 1 P 1 ..., until we arrive at a prediction function that superfi cially seems to fi t our data fairly well. However, that fi t can owe more to this general ability to fi t anything with enough terms than it does to our actual ability to fi nd a small number of valid, reliable factors truly modeling real, underlying processes. Overfi tting is usually considered worst when it in-fl ates Type I error (false statistical signifi cance when none truly exists in the population). Ideally, Type I error should only refl ect sampling error—pure variation due to subject-related factors. In fact, we expect Type I errors about 5% of the time with normally distributed random numbers when the statistical signifi cance level is set at α = .05—because that is precisely how " α = .05 " is defi ned in the fi rst place. But Type I error can also be an unwanted side effect of ill-considered experimental design or statistical method. And this is where this issue of overfi tting relates to our Part I experiment. At some point during the analysis …